ABOUT  STARTING  EARLY  STARTING  SMART 


Startin'*  Marly  Starting  Smart  (SKSS)  is  a  knowledge  development  initiative  designed  to 

•  (Teate  and  test  a  new  model  lor  providin':  integrated  behavioral  health  services  {mental  health  and  substance 
abuse  prevention  and  treatment)  for  votin':  children  (l)irlli  tn  *  years)  and  their  lamilies:  and  to 

•  | idorm  practitioners  and  policymakers  ol  successlul  interventions  and  promising  praetii*cs  from  the  mult i-> * *; 1 1 
study;  which  lay  a  critical  foundation  for  the  positive  growth  and  development  ol  very  youn»-  children. 

In  October  l()97.  with  initial  fundin':  of  $.>()  million,  the  Substance  \hu<e  and  Mental  Health  Services 
Administration  (SAMI  ISA)  and  Casey  Family  Programs  embarked  on  a  precedent-sett  in"  puhlic/private  <*ollahoration. 
twelve  culturally  diverse  grantee  organizations  were  selected.  Kaeh  proxides  integrated  hehaxioral  health  services  in 
community-based  early  childhood  sellin^s-snch  as  Child  Care.  Head  Start  and  IVimarx  Care  Clinies-vv here  yoini" 
families  customarily  receive  services  lor  children.  Critical  to  this  project  is  the  required  collaboration  anion"  hinders, 
orantees.  consumers,  and  local  site  service  proxiders.  Implicit  in  the  desi"ii  ol  this  project  is  Mistaiuabilitv  planning 
for  smirnl  longevity  ol  the  programs. 

The  SKSS  approach  informs  policy-making  lor: 

•  Service  system  redesign  •  Serx ice  access  and  utilization  strategies 

•  St  relict  henin<:  the  home  <mi\ 'ironmenl  *  lar"etin"  benelit"  lor  children 

•  l  I s i n * >  culture  as  a  resource  in  planning  services  •  Working  with  families  from  a  st rciifrt hs-hased 

with  families  perfective 

The  beseareh  Design 

The  \2  "rantees.  vvorkin"  collaborativelv.  designed  a  study  whereby  integrated  behavioral  health  services  are 
delivered  in  typical  early  childhood  setting.  Fach  site  has  an  intervention  and  comparison  "roup,  and  each  site 
delivers  similar  targeted,  culturallv-relevant.  intervention"  lor  vomi"  children  and  their  lamilies.  A  eollabnrativelv 
determined  set  ol  outcomes  has  been  established  to  evaluate  project  ellectiveness: 

•  Access  to  and  use  of  services  •  Care^jxer-ohild  interaction  outcomes 

•  Social,  (‘motional,  and  cognitive  outcomes  lor  •  I  amilx  lunctioniii" 

children 

rrii(‘  "oal  of  the  SKSS  research  is  to  provide  rigorous  scientific  exidence  concerning  whether  ehildnm  and  lamilies 
participating  in  SKSS  programs  achieve  I >et 1 1 “ r  access  to  needed  servin'"  and  better  social,  emotional,  cn"iutive.  and 
behavioral  health  outcomes  than  do  the  ehildnm  and  families  not  receivin':  these  serving.  SKSS  programs  may  also 
"(morale  inlormation  about  opportunities,  pra(*tices.  and  barrieis  to  snu"ht-aller  outcomes.  Ilus  udoi  uial  1011  is 
critical  to  achievin':  effective  public  policies. 

SKSS  Kxlended 

It  was  clear  from  the  early  davs  of  SKSS  that  whatever  effects  were  uncovered,  longitudinal  extension  of  1  he  study 
would  he  valuable.  In  2001.  S \MI  IS  \  and  ( !a>ex  Family  Program"  embarked  upon  an  extension  phase,  which  will 
increase  understanding:  of  the  impact  ol  ealiv  intervention  a"  voiuiii  ehildnm  enter  preschool  and  school  years,  when 
habit's  or  toddlers  are  asked  to  meet  escalatin':  (‘motional  and  cognitive  demands.  This  longitudinal  extension  can 
validate  early  methods  and  findings  and  assc»  their  durability.  It  is  anticipated  that  this  work  will  include  additional 
data  points  of  a  refined  instrument  set  and  intervention  package  with  the  addition  ol  study  questions  relaled  to  cost 
and  value.  and  other  special  studies.  Additional  future  plain  include  applying  and  validating  earlx  SKSS  lessons 
learned.  key  concepts,  components,  and  principles  to  new  settings  that  serve  lamilies  with  vomi"  children. 

Su  munition 

In  sum.  SKSS  reflects  tin*  "rovvui"  acknowledgement  that  it  is  important  to  target  positive  interventions  to  vei  v 
youna  ehildnm.  The  infant  and  preschool  vear>  lav  a  erilieal  foundation  lor  later  "rowt h  and  di'velopment .  Second, 
successful  interventions  for  very  vomi"  children  must  meet  the  multiple  behavioral  health,  physical  health,  and 
educational  needs  of  families.  Third,  integrated  behavioral  In  alt h  services  must  be  made  more  accessible  to  families 
with  multiple  needs,  which  are  diflicull  to  me<‘t  in  a  I  ragmen  ted  service  system. 

Por  more  information  about  Stin  tin'*  Karl\  Starting  Smart  and  related  S \M I  ISA-(  asey  products,  contact 
www-.ciisrv.nrg  or  (SKSS  section  under  construct  ion). 
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PREFACE 


The  increased  interest  in  the  potential  for  early  childhood  intervention  programs  to 
save  dollars  in  the  long  run  has  focused  attention  on  the  potential  for  cost-benefit 
and  related  analyses  to  aid  decisionmakers  in  their  policy  choices.  The  goal  of  this 
report  is  to  identify  the  conceptual  and  methodological  issues  associated  with  the 
analysis  of  costs  and  outcomes  of  early  intervention  programs  in  general  and  to  make 
recommendations  regarding  the  application  of  these  tools  for  subsequent  demon¬ 
stration  studies  of  a  particular  intervention  program:  Starting  Early  Starting  Smart 
(SESS). 

SESS  is  a  public-private  collaboration  designed  to  test  the  effectiveness  of  integrating 
behavioral  health  services  within  primary  care  and  early  childhood  service  settings 
for  children  from  birth  to  age  seven.  The  SESS  program  is  an  initiative  of  the  Office 
on  Early  Childhood,  Substance  Abuse  and  Mental  Health  Services  Administration 
(SAMHSA),  and  the  Casey  Family  Programs,  along  with  several  other  federal  spon¬ 
sors.  The  program  currently  operates  in  12  sites  across  the  United  States  and  is 
entering  the  third  year  of  its  first  five-year  phase.  An  outcomes  evaluation  is  built 
into  the  first  phase. 

Program  sponsors  are  beginning  to  plan  for  a  second  phase,  the  design  of  which  they 
hope  will  be  informed  by  the  first  phase.  It  was  during  the  initiation  of  this  planning 
process  that  program  sponsors  identified  a  need  for  cost  information  to  supplement 
their  outcomes  information.  Recognizing  that  the  literature  offered  somewhat  lim¬ 
ited  guidance  on  the  specifics  of  cost  considerations  in  this  context,  they  requested 
that  RAND  not  only  present  them  with  a  summary  of  research  bearing  on  their 
problem  but  that  we  also  examine  their  program  and  make  specific  recommenda¬ 
tions  regarding  how  cost  and  outcome  analysis  could  improve  their  decisionmaking. 

This  project  began  with  a  meeting  of  cost  and  outcome  analysis  experts  held  in 
August  2000,  convened  by  RAND  on  behalf  of  the  Casey  Family  Programs  and  the 
Office  on  Early  Childhood,  SAMHSA.  Participants  at  the  meeting  included  four 
national  experts  in  cost  and  outcome  analysis  with  backgrounds  in  mental  health 
and  substance  abuse,  as  well  as  several  RAND  staff  members  with  experience  in  cost 
and  outcome  analysis.  Also  participating  were  staff  from  SAMHSA,  the  Casey  Family 
Programs,  the  SESS  Data  Coordinating  Center,  and  two  of  the  SESS  program  sites. 
The  proceedings  from  the  meeting  are  summarized  in  the  following  document: 
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Cannon,  Jill  S.,  Lynn  A.  Karoly,  and  M.  Rebecca  Kilburn,  Directions  for 
Costand  Outcome  Analysis  o/Starting  Early  Starting  Smart:  Summary  of 
a  Cost  Expert  Meeting,  CF-161-TCFP,  Santa  Monica,  California:  RAND, 

2001. 

Readers  interested  in  more  detail  are  urged  to  obtain  a  copy  of  the  full  report,  Assess¬ 
ing  Costs  and  Benefits  of  Early  Childhood  Intervention  Programs:  Overview  and 
Application  to  the  Starting  Early  Starting  Smart  Program,  by  Lynn  A.  Karoly,  M. 
Rebecca  Kilburn,  James  H.  Bigelow,  Jonathan  P.  Caulkins,  and  Jill  S.  Cannon,  which 
can  be  obtained  from  RAND  (www.rand.org),  the  Casey  Family  Programs 
(www.casey.org/projects.htm#sess),  or  SAMHSA  (www.samhsa.gov). 

This  research  is  funded  by  the  Casey  Family  Programs.  The  opinions  expressed  and 
conclusions  drawn  in  this  report  are  the  responsibility  of  the  authors  and  do  not 
represent  the  official  views  of  the  Casey  Family  Programs,  SAMHSA,  other  agencies, 
or  RAND. 
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EXECUTIVE  SUMMARY 


Agency  and  program  administrators  and  decisionmakers  responsible  for  implement¬ 
ing  early  childhood  intervention  programs  are  becoming  more  interested  in  quanti¬ 
fying  the  costs  and  benefits  of  such  programs.  Part  of  the  reason  for  this  is  that 
foundations  and  other  funders  are  putting  more  emphasis  on  results-based  account¬ 
ability.  At  the  same  time,  arguments  for  the  value  of  early  childhood  intervention  are 
being  made  within  the  public  sphere  on  the  basis  of  published  estimates  of  costs  and 
benefits.  Program  implementers  are  naturally  attracted  by  statements  that  a  certain 
intervention  produces  $4  in  savings  for  every  $1  it  costs  and  would  like  to  make  simi¬ 
lar  statements  about  their  own  programs.  Meanwhile,  decisionmakers  without  par¬ 
ticular  interest  in  any  given  program  would  like  more  quantitative  decision  aids 
when  it  comes  time  to  choose  among  a  variety  of  possible  program  models  or  pro¬ 
gram  improvements  to  implement. 

Our  objective  here  is  to  offer  assistance  to  decisionmakers  and  program  imple¬ 
menters  considering  an  assessment  of  costs  and  outcomes.  We  do  not  offer  a  spe¬ 
cific  step-by-step  manual,  but  we  discuss  the  kinds  of  issues  that  must  be  taken  into 
account  and  why.  We  do  so  in  enough  detail  that  readers  can  decide  if  this  type  of 
quantitative  analysis  is  the  right  course  for  them  and,  if  so,  can  knowledgeably  inter¬ 
act  with  an  expert  cost-outcome  analyst.  While  we  understand  that  some  readers 
will  want  to  undertake  analysis  of  costs  and  outcomes  to  justify  a  program  in  which 
they  have  a  special  interest,  we  take  the  viewpoint  here  of  an  unbiased  allocator  of 
funds.  What  evidence  should  such  a  person  want  to  see  before  concluding  that  a 
particular  intervention  is  a  wise  investment?  That  sort  of  evidence  is  what  the 
implementer  seeking  to  justify  further  funding  will  need  to  present. 

We  begin  by  setting  the  conceptual  framework  within  which  program  costs  and  out¬ 
comes  may  be  understood.  We  then  draw  out  some  of  the  implications  of  that  gen¬ 
eral  framework  for  the  analysis  of  early  childhood  interventions  in  particular.  After 
reviewing  some  examples  of  such  analyses,  we  apply  the  methodology  to  an  actual 
case  in  which  a  consortium  of  program  funders  must  decide  whether  to  proceed  with 
an  assessment  and,  if  so,  what  kind  of  assessment  to  undertake.  The  consortium  is 
led  by  the  U.S.  Substance  Abuse  and  Mental  Health  Services  Administration  and  the 
Casey  Family  Programs,  and  the  intervention  of  interest  is  the  Starting  Early  Starting 
Smart  Program. 
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THE  COSTAND  OUTCOME  ANALYSIS  FRAMEWORK 

Decisionmakers  and  program  implementers  just  beginning  to  think  about  analyzing 
costs  and  benefits  are  often  surprised  to  learn  that  several  analytic  avenues  are  open 
to  them.  Which  one  or  ones  they  choose  will  have  important  implications  for  what 
they  learn  and  how  much  they  must  spend  to  learn  it.  Among  the  choices  are  these:1 

•  Cost-benefit  analysis  (or  benefit-cost  analysis)  entails  comparing  a  program’s 
benefits  to  a  stakeholder  with  its  costs  to  that  stakeholder.  Such  a  comparison 
requires  putting  benefits  and  costs  in  comparable  terms,  and  the  terms  conven¬ 
tionally  chosen  are  dollars.  Benefits  that  cannot  be  expressed  in  dollar  terms 
cannot  be  compared  in  this  manner  and  are  included  only  in  associated  qualita¬ 
tive  discussion.  Cost-benefit  analysis  seeks  to  help  in  deciding  whether  a  pro¬ 
gram  is  of  value  to  the  stakeholder.  Often  cost-benefit  analysis  is  conducted 
from  the  perspective  of  society  at  large.2 

•  Cost-savings  analysis  is  restricted  to  the  costs  and  benefits  realized  by  the  gov¬ 
ernment  as  a  whole  or  a  particular  funding  agency.  Only  the  costs  to  the  gov¬ 
ernment  are  taken  into  account,  and  the  benefits  are  those  expressible  as  dollar 
savings  somewhere  in  the  government.  This  kind  of  analysis  is  used  to  deter¬ 
mine  whether  a  publicly  provided  program  “pays  for  itself”  and  is  thus  justified 
not  only  by  whatever  human  services  it  may  render  but  also  on  financial  terms 
alone. 

•  Cost-effectiveness  analysis  determines  how  much  must  be  spent  on  a  program 
to  produce  a  particular  outcome  (or,  what  is  equivalent,  how  much  of  a  particu¬ 
lar  type  of  benefit  will  result  from  a  given  expenditure).  While  this  can  be  done 
for  multiple  outcomes,  no  attempt  is  made  to  sum  the  complete  array  of  benefits 
into  a  single  aggregate  measure. 

•  Cost  analysis  alone  (no  measurement  of  benefits)  can  be  useful  to  decision¬ 
makers  for  a  variety  of  purposes,  for  example,  discovering  which  factors  need  to 
be  considered  in  replicating  a  program  elsewhere  or  for  informing  budget  pro¬ 
jections. 

In  deciding  which  avenues  to  pursue,  the  decisionmaker  or  implementer  must 
choose  what  he  or  she  wishes  to  learn  and  consider  the  funds  available  for  under¬ 
taking  the  analysis.  The  analyses  above  are  ordered  in  terms  of  how  much  attention 
must  be  paid  to  quantifying  outcomes  and  expressing  them  in  dollar  terms  (from  a 
lot  at  the  top  to  none  at  the  bottom).  Other  variables  being  equal,  the  resources  and 
calendar  time  devoted  to  the  analysis  will  drop  with  each  successive  approach  down 
the  list. 


terminology  in  this  field  has  not  been  standardized,  and  these  terms  appear  in  the  literature  with  a  vari¬ 
ety  of  different  meanings.  We  have  chosen  typical  definitions. 

2Of  the  four  analytic  approaches  listed  here,  cost-benefit  analysis  is  subject  to  the  greatest  challenges  in 
execution  and  interpretation.  That  is  because  benefits  must  be  denominated  in  dollars,  and  that  adds 
another  source  of  uncertainty  and  potential  disagreement  over  quantities.  For  some  benefits,  dollar  con¬ 
versions  are  not  really  feasible.  Cost-benefit  assessments  can  thus  rarely  be  comprehensive. 
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As  we  describe  them  here,  these  cost  and  outcome  analysis  methods  are  used  only  as 
components  within  a  broader  decision  support  framework  that  we  call  policy  analy¬ 
sis  or  policy  scorecard  analysis  (the  latter  term  derives  from  the  use  of  a  tool  called  the 
scorecard).3  Despite  the  name,  it  does  not  pertain  only  to  high-level  public  policies 
but  also  to  decisions  made  regarding  specific  strategies  and  programs.  Policy  score- 
card  analysis  offers  a  framework  within  which  to  consider  multiple  benefits,  as 
required  in  the  first  two  approaches  listed  above,  and  multiple  costs,  as  required  by 
all  four.  Policy  scorecard  analysis  also  entails  consideration  of  alternative  programs. 
This  is  important  for  benefit  and  cost  analysis.  In  trying  to  determine  whether  the 
numbers  emanating  from  these  analyses  support  (further)  investment  in  the  pro¬ 
gram,  funders  will  be  asking,  “compared  with  investment  in  what  else?”  A  benefit- 
cost  ratio  of  1.5  to  one  ($1.50  of  benefits  for  every  dollar  of  costs)  may  not  be  good 
enough  if  an  alternative  with  similar  objectives  has  a  ratio  of  two  to  one.  Decision¬ 
makers  will  thus  be  considering  a  range  of  alternative  interventions  or  at  least  a 
choice  between  funding  the  program  in  question  and  some  default  course  of  action 
(which  could  be  leaving  things  as  they  are). 

The  results  of  a  policy  scorecard  analysis  can  be  summarized  in  a  simple  tool  called  a 
scorecard.  The  scorecard  lists  benefit  and  cost  categories  down  the  side,  together 
with  program  design  features  influencing  them,  and  the  alternative  courses  of  action 
across  the  top.  Thus,  each  cell  in  the  scorecard  gives  a  particular  cost  or  benefit  (or 
design  feature)  for  a  particular  program.  In  identifying  the  row  and  column  heads 
and  filling  in  the  cells — that  is,  in  conducting  the  policy  scorecard  analysis — several 
guidelines  must  be  kept  in  mind: 

•  Designate  which  benefits  and  costs  accrue  to  which  stakeholders.  If  you  say 
that  a  program  generates  more  savings  than  costs,  people  will  want  to  know,  sav¬ 
ings  to  whom?  And  costs  to  whom? 

•  Define  explicitly  the  period  over  which  the  analysis  applies.  If  the  purpose  of 
the  analysis  is  to  determine  whether  a  program  has  a  favorable  benefit-cost  ratio 
or  pays  for  itself  in  government  savings,  it  is  better  to  look  well  into  the  future. 
No  one  period  or  duration  is  correct,  however.  The  choice  depends  on  the 
patience  of  the  decisionmakej;  in  question,  with  individuals  typically  having 
shorter  planning  horizons  than  society  as  a  whole.  This  distinction  makes  a  dif¬ 
ference  because  the  costs  of  early  intervention  programs  typically  accrue  over  a 
matter  of  months  or  a  few  years,  whereas  the  benefits  are  often  not  fully  realized 
until  the  participating  children  age  into  adulthood.  Counting  such  benefits 
directly  entails  long-term  follow-up  of  program  subjects,  though  some  future 
benefits  can  be  predicted  on  the  basis  of  shorter-term  trends. 

•  Discount  future  costs  and  benefits.  Although  it  is  important  to  count  future 
benefits  (and  costs),  they  cannot  be  counted  at  full,  nominal  value.  People  dis¬ 
count  future  benefits  and  costs:  getting  a  $1,000  benefit  five  years  in  the  future 


3The  term  policy  analysis  was  originally  adopted  by  RAND  analysts  and  others  to  describe  an  approach  for 
quantitatively  analyzing  management  problems.  Today,  the  term  is  used  even  more  broadly  to  charac¬ 
terize  a  wide  range  of  quantitative  and  qualitative  approaches  to  addressing  policy  issues.  Hence,  we  will 
employ  the  more  focused  term  policy  scorecard  analysis  for  the  remainder  of  this  summary. 
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does  not  look  as  attractive  as  getting  it  now;  having  to  pay  $1,000  five  years  in  the 
future  does  not  seem  as  onerous  as  having  to  pay  it  now.  A  real  annual  discount 
rate  of  3  percent  to  6  percent  is  typically  applied  to  future  benefits  and  costs. 

•  Record  cost  elements  as  resource  quantities.  Until  the  figures  are  added  up  at 
the  end,  costs  should  be  recorded  in  terms  of  resource  quantities — hours  of 
labor,  square  footage  of  rental  space,  etc. — rather  than  in  dollar  terms.  Prices  for 
these  resources  can  vary  from  one  site  to  another,  and  on-budget  dollars  in  par¬ 
ticular  do  not  always  reflect  total  costs.  A  physician  may  donate  time  on  the 
weekends,  but  from  society's  point  of  view,  that  time  is  not  “free”;  perhaps  it 
could  have  been  put  to  another,  more  beneficial  use. 

•  Address  uncertainty.  Future  benefits  and  costs  cannot  often  be  predicted  with 
great  confidence.  Where  a  range  of  values  is  plausible,  that  range  should  be 
made  explicit  in  the  analysis.  Likewise,  structural  uncertainty  (e.g.,  about  possi¬ 
ble  future  changes  in  laws  relevant  to  a  program)  should  also  be  considered. 

The  final  step  in  the  cost  and  outcome  analysis  is  to  add  up  all  the  benefits  (or  sav¬ 
ings)  and  add  up  all  the  costs  and  compare  them  across  programs.  The  four  methods 
listed  above  offer  alternative  ways  for  performing  this  step.  Cost-benefit  and  cost- 
savings  analysis  each  provide  a  single  measure  of  merit  for  each  alternative;  the 
alternative  with  the  greatest  merit  according  to  this  measure  is  declared  the  winner. 
Cost-effectiveness  analysis  provides  multiple  measures  of  merit.  They  can  be  com¬ 
bined  into  a  single  measure  (e.g.,  the  ratio  of  effectiveness  to  cost,  if  a  single  effec¬ 
tiveness  measure  dominates),  which  will  be  used  in  the  same  way  as  a  cost-benefit  or 
cost-savings  measure.  Or  they  can  be  used  to  define  a  different  kind  of  selection  rule, 
one  that  deems  “best”  the  policy  that  achieves  a  specified  level  of  effectiveness  at 
lowest  cost  (a  constant  effectiveness  analysis)  or  that  achieves  the  greatest  effective¬ 
ness  for  a  given  cost  (a  constant  cost  analysis).4 

Comparing  costs  and  benefits  may  not  produce  a  single  “answer”  that  one  program 
is  obviously  preferable  to  another.  One  program  may  produce  a  net  benefit  to  one 
group  of  stakeholders,  while  another  benefits  a  second  group.  The  net  benefit  of  one 
program  may  be  somewhat  higher  than  that  for  another,  but  the  uncertainty  ranges 
may  overlap  so  much  that  the  advantage  cannot  be  asserted  with  high  confidence. 
Some  possible  change  in  the  institutional  environment,  e.g.,  tax  reform,  could  shift 
benefits  and  costs  enough  to  change  the  advantage  from  one  program  to  another. 
Such  possibilities  would  not  subtract  from  the  value  of  the  cost  and  outcome  analy¬ 
sis.  On  the  contrary,  some  of  the  most  valuable  insights  are  suggestions  for  policy 
changes  that  reallocate  benefits  across  stakeholder  groups  so  that  all  of  them  gain 
and  thus  have  no  incentive  to  block  a  program. 

In  most  studies,  the  majority  of  the  analytical  effort  will  come  from  learning  about 
the  domain,  structuring  the  models  of  how  the  intervention  works,  collecting  and 
cleaning  data,  etc.  In  short,  filling  in  the  scorecard  is  challenging.  Given  that 


4Ihe  latter  is  sometimes  called  a  constant  budget  analysis,  but  this  is  only  appropriate  if  all  the  costs 
appear  in  the  budget  of  the  agency  making  the  decision.  In  many  programs,  costs  may  be  distributed 
across  many  stakeholders.  They  will  not  ail  appear  in  any  single  party’s  budget. 
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groundwork,  computing  the  summary  evaluation  metrics  is  straightforward,  whether 
that  metric  is  a  benefit-cost  or  a  cost-effectiveness  ratio. 

Hence,  instead  of  suggesting  that  one  must  choose  to  implement  one  of  these  four 
approaches,  it  is  more  accurate  to  say  that  one  must  choose  whether  or  not  to  con¬ 
duct  a  careful,  quantitative  summation  of  the  effects  of  the  program.  If  the  answer  is 
yes,  then  there  follows  a  choice  of  how  one  is  going  to  present  the  results  of  that 
analysis  to  decisionmakers,  as  a  benefit-cost  ratio,  cost-effectiveness  ratio,  and  so  on, 
or  some  combination  thereof. 

It  is  thus  important  to  keep  cost-benefit  analysis,  cost-savings  analysis,  and  other 
forms  of  cost  and  outcome  analysis  in  their  place.  In  any  decision,  some  factors  can 
be  resolved  only  through  a  decisionmaker’s  values  and  subjective  judgment  or 
through  negotiation  among  stakeholders.  Likewise,  the  public  quantifying  of  deci¬ 
sion  factors  may  occasionally  be  problematic  (e.g.,  when  an  auto  manufacturer  com¬ 
pares  the  cost  of  a  safety  improvement  with  the  dollar-equivalent  benefit  of  the  lives 
that  could  be  saved  by  that  design  change).  Nevertheless,  these  methods  can  provide 
valuable  input  to  choosing  among  different  programs,  demonstrating  a  program’s 
worth,  improving  programs,  and  replicating  them. 

APPLYING  THE  FRAMEWORK  TO  EARLY  CHILDHOOD  INTERVENTIONS 

Early  intervention  programs  attempt  to  improve  child  health  and  development  by 
providing  young  children  and  their  families  various  social  services  and  supports. 
Such  programs  can  have  effects  in  four  domains:  emotional  and  cognitive  develop¬ 
ment,  education,  economic  well-being  (in  terms  of  public  assistance,  income,  and 
crime),  and  health.  Specific  examples  of  possible  benefits  within  each  of  these  cate¬ 
gories  are  given  in  Table  S.l.  Which  benefits  are  measured  depends  on  the  purpose 
of  the  analysis.  Cost-benefit  and  cost-savings  analyses  typically  seek  a  comprehen¬ 
sive  accounting  of  the  benefits  to  society  or  to  government  (respectively),  although 
many  benefits  are  difficult  to  express  in  dollar  terms  and  therefore  cannot  be  aggre¬ 
gated  in  the  cost-benefit  assessment.  While  cost-effectiveness  analysis  can  in  princi¬ 
ple  be  performed  for  any  outcome,  it  is  often  the  case  in  practice  that  a  single  benefit 
or  a  narrow  set  receives  most  of  the  attention.  A  full  analysis  of  the  benefits  of  an 
early  intervention  program  should  include  collection  of  data  on  as  many  potential 
benefits  as  the  analyst’s  resources  permit. 

Note  that  early  childhood  interventions  can  benefit  parents  and  other  caregivers 
while  simultaneously  helping  children.  It  is  important  to  measure  benefits  to  care¬ 
givers,  because  these  are  often  realized  over  much  shorter  time  periods  than  are 
those  accruing  to  children.  Ignoring  these  benefits  means  underestimating  a  pro¬ 
gram’s  benefit-cost  ratio  or  its  potential  net  savings  to  government,  particularly  over 
the  short  term — and  for  some  analyses,  it  will  only  be  feasible  to  make  short-term 
measurements. 

Any  analysis  of  benefits  of  a  program  under  way  must  include  a  comparison  group. 
This  is  a  group  of  children  and  caregivers  not  enrolled  in  the  program  but  similar  in 
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Table  S.l 

Early  Childhood  Intervention  Program  Benefit  Domains 
and  Illustrative  Measures 


Illustrative  Measures  for: 

Benefit  Domain 

Child 

Parent/Caregiver 

Emotional  and  cognitive 

Sociocmotional  and 

Quality  of  parent-child 

development 

behavior  scores 

IQ  test  scores 

Teacher’s  ratings 

relationship 

Quality  of  home  envi¬ 
ronment 

Education 

Achievement  test  scores 
Grades 

Grade  progression 
(repetition) 

Participation  in  special 
education 

Educational  attainment 

Educational  attainment 

Public  assistance  receipt, 

Receipt  of  public 

Receipt  of  public 

income,  crime 

assistance 

Employment 

Earnings/income 

Criminal  activity 

Contact  with  criminal 
justice  system 

assistance 

Employment 
Earnings/income 
Criminal  activity / 
Contact  with  criminal 
justice  system 

Health 

Physical  and  mental 
health  status 

Child  abuse  and  neglect 
Substance  abuse 

Fertility  control 
Emergency  room  visits 
Other  health  care  use 

Physical  and  mental 
health  status 

Family  violence 
Substance  abuse 
Fertility  control 

NOTE:  Italics  indicate  measures  more  easily  expressed  in  dollar  terms. 


as  many  ways  as  possible  to  the  program  participants  and  whose  progress  along  the 
various  benefit  measures  is  tracked.5  Children  in  particular  have  a  tendency  to 
improve  along  various  measures  of  development  as  they  grow.  Evaluators  must  take 
care  to  ensure  that  the  program  benefits  they  measure  are  net  of  what  would  have 
occurred  naturally  or  what  children  would  realize  anyway  from  outside  influences 
without  the  program.  Measurements  of  the  comparison  group  provide  estimates  of 
benefits  that  would  have  accrued  in  the  program's  absence. 

Data  on  progress  along  benefit  measures  can  be  collected  by  survey  questionnaires, 
tests,  or  other  means  of  direct  interaction  with  the  children  and  their  caregivers.  For 
some  benefit  types  (e.g.,  reductions  in  involvement  with  the  criminal  justice  system), 
administrative  data  may  be  available.  When  only  a  few  years  of  data  collection  are 
feasible,  a  glimpse  into  the  future  can  be  obtained  through  mathematical  models 


5Idea11y,  one  should  randomly  assign  children  and  caregivers  to  program  participation  versus  the  com¬ 
parison  group.  This  ensures  that  the  participation  and  comparison  groups  are  (statistically)  identical  in 
both  measured  and  unmeasured  characteristics.  When  the  comparison  group  is  selected  by  random 
assignment,  it  is  often  called  a  control  group.  When  random  assignment  is  not  feasible  or  desirable,  a 
comparison  group  can  still  be  chosen,  by  identifying  children  and  caregivers  who  are  similar  in  various 
measured  ways  to  the  program  participants. 
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that  can  predict  future  criminal  activity  or  future  earnings  on  the  basis  of  childhood 
information.  (This  cannot  of  course  be  done  with  confidence  for  any  given  child,  but 
results  obtained  for  a  group  of  children  may  be  sufficiently  reliable  for  the  purpose.) 

As  with  benefits,  the  cost  elements  to  be  included  in  an  analysis  depend  on  its  pur¬ 
pose.  For  example,  costs  that  accrue  to  society  but  not  to  a  funding  agency  are 
included  in  a  societal  cost-benefit  analysis  but  omitted  from  a  cost-savings  analysis. 
Regardless  of  the  analysis  to  be  performed,  program  costs  must  be  estimated  as  net 
of  those  accrued  by  comparison  group  children  for  similar  services.  For  example,  if 
an  intervention  is  intended  to  increase  prenatal  care,  the  analysis  should  include 
only  the  resources  devoted  to  the  visits  and  services  received  by  program  participants 
in  excess  of  what  they  would  have  received  anyway  (i.e.,  in  excess  of  those  received 
by  the  comparison  group). 

Estimation  of  costs  should  follow  the  general  guideline  given  above  regarding  the 
need  to  estimate  resource  quantities  instead  of  dollars  and  to  account  for 
“opportunity”  costs  and  other  off-budget  resource  expenditures.  Costs  borne  by 
participants  should  also  be  included,  as  well  as  costs  borne  by  other  agencies  or  ser¬ 
vice  providers.  Collecting  cost  data  for  the  same  set  of  service  providers  for  both  the 
treatment  and  control  groups  allows  the  analyst  to  detect  both  cost  shifting  (e.g., 
from  one  payor  to  another)  and  cost  offsets  (e.g.,  reduced  utilization  of  services  in 
one  area  as  a  result  of  increased  service  use  in  another).  It  may  also  be  useful  to  dis¬ 
tinguish  between  the  fixed  costs  of  implementing  a  program  that  are  not  dependent 
on  the  number  of  children  served  and  the  variable  costs  that  are.  The  split  between 
fixed  and  variable  costs  will  influence  the  calculation  of  benefit-cost  ratios,  net  sav¬ 
ings,  and  cost-effectiveness  ratios  for  programs  when  scaled  up  to  serve  larger  num¬ 
bers  of  children. 

SOME  ILLUSTRATIVE  ANALYSES 

Given  the  challenges  and  requirements  outlined  so  far,  it  should  not  be  surprising 
that  not  many  scientifically  sound  cost-benefit  and  cost-savings  analyses  of  early 
childhood  intervention  programs  with  long-term  follow-ups  have  been  conducted. 
Among  those  recently  analyzed  or  reanalyzed  are  the  following: 

•  The  Perry  Preschool  program  provided  center-based  classes  and  teacher  home 
visits  for  one  or  two  school  years  to  58  children  ages  three  or  four  in  Ypsilanti, 
Michigan,  from  1962  to  1967.  Benefits  were  tracked  for  both  the  participants  and 
the  comparison  group  (65  children)  through  age  27.  Benefits  included  better 
school  performance,  higher  employment,  less  welfare  dependence,  and  lower 
involvement  in  criminal  activity  on  the  part  of  participants.  The  most  recent 
cost-benefit  assessment  evaluates  benefits  expressible  in  monetary  terms  at 
$50,000  per  child,  half  of  that  in  the  form  of  savings  to  government,  versus  a  pro¬ 
gram  cost  of  $12,000  per  child  (see  Figure  S.l). 

•  In  the  Prenatal/Early  Infancy  Project  (PEIP)  in  Elmira,  New  York,  nurses  started 
visiting  mothers  when  they  were  pregnant  and  continued  until  their  child  was 
age  two.  The  objective  was  to  improve  pregnancy  outcomes  and  parenting  skills 
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and  link  the  mother  with  social  services.  Between  1978  and  1980  the  program 
reached  116  first-time  mothers.  They  and  another  184  in  the  control  group  have 
been  followed  through  age  15  of  the  first-born  child.  Benefits  for  the  mothers 
included  better  pregnancy  behaviors  and  less  child  abuse  in  the  short  term  and 
lower  welfare  participation  and  criminal  behavior  in  the  long  term.  The  children 
benefited  as  well  in  several  domains.  For  the  higher-risk  portion  of  the  sample 
(unmarried  mothers  with  low  socioeconomic  status),  benefits  amounted  to 
almost  $31,000  per  mother-child  pair,  with  almost  half  of  that  in  the  form  of  a 
reduction  in  welfare  received  by  the  mother.  For  the  lower-risk  portion  of  the 
sample,  however,  benefits  came  to  only  $6,700.  Program  costs  were  about 
$6,100. 

•  The  Chicago  Child-Parent  Centers  have  promoted  reading  and  language  skills, 
provided  health  and  social  services,  and  promoted  parent  involvement  for  chil¬ 
dren  in  preschool  through  third  grade.  A  cohort  of  989  children  completing 
kindergarten  in  1986  was  tracked  to  age  20  and  compared  with  a  no-preschool 
group  of  550  children.  The  program  resulted  in  long-lasting  educational- 
achievement  benefits.  Higher  between-grade  promotion  rates,  reduced  special- 
education  use,  increased  earnings  expected  as  a  result  of  better  educational  per¬ 
formance,  and  lower  involvement  with  the  juvenile  justice  system  translated  into 
about  $35,000  in  benefits  per  program  participant.  The  program  cost  nearly 
$10,000  per  participant. 
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Figure  S.l — Some  Early  Childhood  Interventions  Have  Been  Shown  to  Have 
High  Benefit-Cost  Ratios 
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These  analyses  demonstrate  that  early  childhood  interventions  can  generate  savings 
to  government  and  benefits  to  society  that  exceed  program  costs.  Indeed,  for  most  of 
the  samples  reported  above,  benefits  were  a  multiple  of  costs,  and  all  of  these  pro¬ 
grams  resulted  in  benefits  that  could  not  be  translated  into  costs  and  were  thus 
omitted.  Therefore,  decisionmakers  and  implementers  thinking  about  performing 
analyses  of  costs  and  benefits  should  not  give  up  merely  because  they  don’t  see  how 
some  of  a  program’s  principal  benefits  can  be  converted  to  dollar  terms. 

Two  further  lessons  for  cost-benefit  analysis  may  be  drawn  from  these  examples. 
First,  many  important  benefits  can  only  be  captured  through  an  extended  time  hori¬ 
zon.  The  savings  from  Perry  Preschool,  for  example,  did  not  accumulate  to  match 
the  level  of  program  costs  until  the  participants  were  20  years  old.  Some  of  these 
benefits  can  be  predicted  on  the  basis  of  shorter  trends,  but  not  all  can,  and  confi¬ 
dence  in  predicted  results  increases  as  follow-up  periods  lengthen. 

Second,  programs  can  be  beneficial  to  caregivers  as  well  as  to  children.  In  fact,  when 
time  is  lacking  for  lengthy  follow-ups  or  when  they  are  not  feasible,  measuring  bene¬ 
fits  to  caregivers  can  result  in  early  favorable  benefit-cost  ratios  and  net  savings.  The 
Elmira  program  was  the  only  one  of  those  summarized  that  measured  caregiver 
benefits,  and,  in  that  case,  savings  sufficient  to  balance  costs  were  tallied  within  two 
years  of  the  end  of  program  services. 


FRAMING  A  POLICY  SCORECARD  ANALYSIS  FOR  A  SPECIFIC  PROGRAM 

The  Starting  Early  Starting  Smart  (SESS)  program  is  intended  to  test  the  effectiveness 
of  integrating  mental  health  services  and  substance  abuse  prevention  and  treatment 
into  early  childhood  education  or  primary  health  care  for  children  from  birth  to  age 
seven.  The  program  is  under  way  at  12  sites  nationwide,  seven  using  the  early  child¬ 
hood  (EC)  education  model  and  five  using  the  primary  care  (PC)  paradigm.  (See  the 
appendix  of  the  full  report  [forthcoming]  for  a  description  of  each  state.)  Most  of  the 
sites  serve  between  100  and  300  children,  and  comparison  groups  average  out  to 
similar  numbers. 

By  “effectiveness,”  the  program  means  increased  access  to,  use  of,  and  satisfaction 
with  behavioral  health  services  and  increased  social,  emotional,  and  cognitive  func¬ 
tioning  on  the  part  of  served  children.  Data  on  these  benefit  measures  are  being 
collected  over  an  18-month  follow-up  period  at  intervals  that  average  six  months  (PC 
sites)  or  nine  months  (EC  sites).  No  cost  data  are  being  gathered  in  this  first  phase  of 
the  program,  but  a  second  phase  is  being  planned,  and  part  of  that  planning  is  to 
assess  the  feasibility  of  cost  and  outcome  analysis. 

SESS  program  implementers  are  wise  to  take  cost  and  benefit  evaluation  issues  into 
account  in  the  planning  stage.  Too  often,  evaluation  is  considered  only  after  pro¬ 
gram  design  has  been  finalized  along  lines  that  preclude  sound  cost  and  benefit 
assessment.  SESS’s  Phase  I  design  raises  issues  that  need  to  be  resolved  for  Phase  II 
if  cost  and  outcome  analyses  are  to  be  possible.  One  issue,  for  example,  is  that  some 
sites  did  not  use  random  assignment  (primarily  EC  sites),  which  raises  concerns 
about  the  validity  of  the  treatment  group  versus  comparison  group  difference  as  a 
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measure  of  the  true  effects  of  the  program.  Future  demonstration  sites  should  aim 
for  random  assignment  if  at  all  possible.  Another  concern  is  that  a  few  sites  are 
experiencing  relatively  high  dropout  rates,  which  could  bias  benefit  estimates  if  those 
who  are  lost  to  follow-up  are  different  from  those  who  remain  in  the  study  and  if  they 
differ  in  important  ways  that  cannot  be  observed.  Obtaining  a  consistently  high  fol¬ 
low-up  rate  across  sites  would  need  to  be  a  priority  in  Phase  II.  Also,  Phase  I  has 
been  characterized  by  between-site  variations  in  services.  This  is  problematic  from 
an  evaluation  standpoint  for  a  couple  of  reasons:  It  complicates  interpretation  of 
results,  and  it  complicates  the  design  of  comparison  groups. 

The  design  of  comparison  groups  for  SESS  offers  lessons  for  other  programs. 
Because  SESS  attempts  to  integrate  behavioral  health  services  into  existing  early 
childhood  and  primary-care  settings,  only  the  benefits  of  the  new,  integrated  services 
plus  increases  in  the  “dosages”  of  existing  services  may  be  credited  to  SESS,  not  the 
full  benefits  realized  from  participation  in  the  early  childhood  program  and  primary 
care.  Similarly,  only  the  costs  associated  with  these  incremental  activities  should  be 
considered.  Therefore,  the  comparison  groups  must  be  designed  to  isolate  the  SESS 
effects  by  including  everything  except  SESS.  The  appropriate  comparison  groups  for 
this  evaluation  would  consist  of  children  involved  in  early  childhood  and  primary- 
care  programs  without  the  integrated  SESS  services,  not  children  receiving  no  ser¬ 
vices  at  all. 

In  the  policy  analysis  scorecard,  then,  the  columns  would  correspond  to  the  early 
childhood  program  without  SESS,  primary-care  program  without  SESS,  and  then  the 
integrated  EC  plus  SESS  and  PC  plus  SESS  interventions,  along  with  whatever  vari¬ 
ants  are  retained.  The  rows  would  be  the  program  descriptors  and  cost  and  benefit 
categories.  The  program  features  reported  would  be  those  having  implications  for 
costs  or  benefits,  e.g.,  population  served,  eligibility  criteria,  age  of  children  at  enroll¬ 
ment,  qualifications  of  program  personnel,  types  and  “dosages"  of  services  rendered, 
transportation  provisions,  and  so  on.  In  future  demonstrations,  this  information  can 
be  collected  through  site  visits  and  other  mechanisms  currently  being  used  in  the 
evaluation  of  Phase  I. 

Cost  estimates  would  begin  with  the  cost  of  serving  one  child  (or  child’s  caregiver)  in 
terms  of  labor  hours  expended  with  the  child  and  in  preparing  for  the  session  and  in 
terms  of  materials  consumed.  These  would  then  be  multiplied  by  dosage  per  child 
and  number  of  children  served.  Fixed  costs  unrelated  to  number  of  children  served, 
such  as  space  rental,  would  then  be  identified.  Multiplication  by  unit  costs  to  con¬ 
vert  to  dollars  would  be  done  last.  Ultimately,  the  cost  information  should  be  as 
comprehensive  as  possible  and  comparable  across  demonstration  sites. 

Benefit  measures  now  being  collected  for  SESS  include  information  on  child  problem 
behavior  and  social  skills,  child  cognitive  development,  parent-child  interaction, 
caregiver  stress  and  negative  or  positive  behaviors,  caregiver  mental  health  prob¬ 
lems,  caregiver  education  and  employment,  and  home  environment.  As  discussed 
above,  the  emphasis  on  both  child  and  caregiver  benefits  will  be  important  to  making 
the  short-run  benefit  tally  as  complete  as  possible.  Almost  all  of  these  measures, 
however,  are  within  the  domain  of  emotional  and  cognitive  development  and  are  not 
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easily  expressed  in  dollar  terms.  This  makes  a  formal  cost-benefit  or  cost-savings 
analysis  problematic  in  that  only  a  limited  set  of  outcomes  might  possibly  be  valued 
in  dollar  terms  to  be  compared  with  program  costs.  Unless  the  program  impact  for 
those  outcomes  valued  in  dollar  terms  is  very  large  and  favorable,  so  that  sizable 
dollar  benefits  are  generated,  a  cost-benefit  analysis  would  be  unlikely  to  show  a 
favorable  outcome  for  the  SESS  program  based  on  the  information  available  after 
two  years. 

While  not  the  program’s  main  intent,  other  benefits  could  result  from  it.  Some  of 
these  benefits,  in  such  areas  as  physical  health,  labor  market  outcomes,  and 
involvement  with  the  criminal  justice  system,  could  be  more  easily  expressed  in  dol¬ 
lar  terms  than  those  now  being  measured.  These  outcomes  could  be  collected  for 
parents  or  caregivers  in  the  short  term,  and  with  longer-term  follow-up,  for  the  par¬ 
ticipating  children.  If  behavioral  changes  are  large  in  these  areas  as  a  result  of  the 
SESS  intervention,  they  can  produce  sizable  dollar  benefits  that,  even  when  dis¬ 
counted,  will  be  a  large  offset  to  the  costs  of  the  program.  This  is  especially  relevant 
for  changes  in  parental  behavior  that  can  be  measured  even  in  the  short  run. 
Improvements  of  adult  economic  and  health  outcomes  have  been  demonstrated  to 
produce  substantial  short-run  benefits  in  other  early  childhood  programs. 

Costs  and  outcomes  would  be  measured  for  both  the  participant  and  comparison 
groups,  with  the  difference  between  the  two  constituting  the  incremental  cost  and 
benefits  from  implementing  SESS.  To  compare  the  present  values  of  all  costs  and 
benefits,  it  will  be  important  to  predict  how  they  will  accrue  over  time.  Costs  and 
benefits  should  also  be  categorized  according  to  which  groups  incur  them.  It  will  be 
of  interest,  for  example,  to  know  how  much  the  intervention  costs  and  benefits  par¬ 
ticipants,  the  agency  implementing  the  program,  other  agencies,  and  society  as  a 
whole. 

Taking  all  these  steps  would  be  sufficient  to  support  as  full  a  cost-benefit  or  cost- 
savings  analysis  as  is  likely  to  be  feasible  given  the  current  state  of  the  art.  If  SESS 
decisionmakers  wish  to  be  able  to  say  something  about  the  value  the  program 
returns  to  society  relative  to  its  costs,  the  preceding  array  of  evaluation  tasks  and 
program  design  modifications  would  be  required.  If  they  decide  it  is  enough  to  be 
able  to  say  how  much  the  program  saves  the  government  relative  to  what  it  costs, 
then  some  elements — costs  to  participants  or  losses  to  crime  victims,  for  example — 
can  be  omitted.  The  overall  level  of  effort  required,  however,  is  not  likely  to  change 
very  much. 

If  SESS  funders  or  implementers  would  like  instead  to  focus  on  one  or  a  few  promi¬ 
nent  measures  of  effectiveness  to  compare  the  different  SESS  variants  with  each 
other,  a  cost-effectiveness  analysis  should  be  sufficient.  By  collecting  cost  data, 
along  with  data  on  that  one  or  those  few  benefits,  it  would  be  possible  to  say,  for 
example,  how  much  child  problem  behavior  decreased  (relative  to  no  SESS)  per 
thousand  dollars  spent  on  SESS  plus  EC  or  SESS  plus  PC.  No  conversion  of  the  ben¬ 
efit  to  dollar  terms  would  be  necessary. 

Finally,  if  the  purpose  was  to  find  out  how  much  program  modifications  or  prolifera¬ 
tion  of  sites  would  cost,  no  benefit  data  would  be  necessary  at  all.  Clearly,  program 
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decisionmakers  may  have  to  make  trade-offs  between  what  they  might  like  to 
achieve  and  how  much  of  a  resource  commitment  they  are  willing  or  able  to  make. 

CONCLUSIONS 

The  recommendations  we  offer  specific  to  the  SESS  program  may  be  framed  as  a  set 
of  more-general  guidelines  for  decisionmakers  considering  cost  and  outcome  analy¬ 
sis  of  an  early  childhood  intervention  program.  In  particular,  among  the  recommen¬ 
dations  that  can  be  applied  more  broadly  are  the  following: 

•  Regarding  the  design  of  a  program  evaluation  and  cost  and  outcome  analysis: 

—  Specify  the  explicit  goals  of  the  cost  and  outcome  analysis  to  guide  the  scope 
of  cost  and  benefit  data  collection  and  analysis. 

—  Identify  comparison  groups  and  track  the  same  cost  and  outcome  measures 
for  both  comparison  and  participant  groups.  If  possible,  use  random 
assignment  to  define  comparison  groups  to  provide  a  more  valid  test  of 
intervention  program  effects. 

—  To  minimize  attrition  in  a  longitudinal  study,  devote  resources  to  retaining 
study  subjects. 

—  Collect  information  on  program  features  through  site  visits  and  other  mecha¬ 
nisms  to  accurately  characterize  features  of  the  intervention  models  as  they 
are  implemented  and  to  ensure  fidelity  to  the  program  model. 

•  Regarding  the  collection  and  analysis  of  cost  data: 

—  Collect  cost  information  for  both  treatment  and  comparison  groups  at  each 
site  where  the  intervention  program  is  implemented. 

—  Ensure  that  the  cost  information  is  as  comprehensive  as  possible:  Costs 
borne  by  various  parties  should  be  differentiated,  the  period  in  which  costs 
are  incurred  should  be  identified,  and  direct  and  indirect  costs,  fixed  and 
variable  costs,  and  goods  and  services  provided  in-kind  should  be  measured. 

—  Plan  for  proper  training  and  technical  support  of  implementation  sites  and 
any  cross-site  data  collection  organizations  to  ensure  uniformity  in  the  col¬ 
lection  of  cost  data.  Collect  information  on  the  cost  of  data  collection, 
training  and  support,  and  the  related  analyses  of  the  data. 

•  Regarding  the  collection  and  analysis  of  outcome  data: 

—  If  cost-benefit  or  cost-savings  analysis  is  the  goal,  include  in  the  outcome 
data  information  for  parents  and  other  caregivers  in  the  short  term  and  long 
term  and  for  children  in  the  long  term  in  those  domains  with  outcomes  that 
can  be  readily  evaluated  in  terms  of  dollars  and  can  produce  large  dollar 
benefits.  The  choice  of  specific  outcome  measures  should  be  guided  by 
findings  from  related  evaluation  studies  whenever  possible. 
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—  Obtain  information  from  participants  that  facilitates  collection  of  adminis¬ 
trative  data  and  allows  effective  tracking  of  individuals  to  increase  response 
rates  at  later  follow-ups. 

—  Where  possible,  collect  complete  histories  using  retrospective  survey  ques¬ 
tions  or  administrative  data  for  outcomes  that  may  generate  a  continuous 
flow  of  dollar  benefits  (e.g.,  labor  market  outcomes,  social  welfare  program 
use,  use  of  costly  health  or  education  services). 

—  When  supported  by  other  empirical  evidence,  project  future  benefits  based 
on  observed  outcomes.  Consider  additional  method  development  that 
would  permit  such  forecasts  for  a  broader  range  of  outcomes. 

While  we  believe  these  principles  are  quite  general,  ultimately  these  recommenda¬ 
tions  should  be  viewed  as  guidelines  that  may  need  to  be  tailored  to  the  specific  cir¬ 
cumstances  of  a  given  intervention  program  and  its  evaluation  design.  In  the  end, 
the  objectives  of  a  program’s  decisionmakers  will  dictate  the  shape  of  the  analysis. 

The  general  policy  scorecard  analysis  tools  considered  in  this  report,  and  those  spe¬ 
cific  to  cost  and  outcome  analysis,  have  great  promise  for  improving  decisionmaking 
with  respect  to  such  investment  programs  as  the  early  childhood  interventions  rep¬ 
resented  by  SESS  and  its  counterparts.  When  used  with  skill  and  judgment,  the 
application  of  these  methods  to  other  programs,  such  as  SESS,  will  further  broaden 
our  base  of  knowledge  regarding  the  value  of  these  investments  and  aid  decision¬ 
makers  in  their  choice  among  program  alternatives. 
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The  SESS  program  is  an  initiative  of  the  Office  on  Early  Childhood,  Substance  Abuse, 
and  Mental  Health  Services  Administration  (SAMHSA)  and  the  Casey  Family  Pro¬ 
grams,  along  with  other  federal  sponsors.  Patricia  Salomon,  Director  of  the  Office  of 
Early  Childhood  at  SAMHSA,  oversees  the  SESS  program  along  with  project  officers 
Michele  Basen,  Velva  Spriggs,  and  Jocelyn  Whitfield,  and  staff  Shakeh  Kaftarian.  At 
the  Casey  Family  Programs,  the  partnership  is  overseen  by  Jean  McIntosh  and 
Barbara  Kelly-Duncan,  along  with  project  officers  Eileen  O’Brien  and  Peter  Pecora. 

The  SESS  program  currently  operates  in  12  sites  across  the  U.S.  Table  A.l  lists  each 
of  the  study  sites  and  the  associated  principal  investigator,  project  director,  and  local 
researcher,  first  for  the  primary  care  (PC)  sites  and  then  for  the  early  childhood  (EC) 
sites.1  Information  about  the  Data  Coordinating  Center  is  also  provided  in  Table  A.l. 
A  brief  description  of  the  program  at  each  site  is  provided  in  the  appendix  to  the 
companion  report.  Further  information  about  the  SESS  program  is  provided  in 
Appendix  B  and  Appendix  C  and  is  available  from  the  Casey  Family  Programs  (www. 
casey.org/projects.htm#sess)  and  SAMHSA  (www.samhsa.gov). 

Table  A.l 
SESS  Grant  Sites 


Study  Site 

Principal 

Investigator 

Project  Director 

Local  Researcher 

Data  Coordinating  Center 

EMT  Associates,  Inc.,  Folsom, 
Calif.,  (615)  595-7658 

Joel  Phillips 

J.  Fred  Springer, 
Ph.D. 

J.  Fred  Springer, 

Ph.D. 

Primary  Care  Sites 

Boston  Medical  Center,  Boston, 
Mass.,  (617)  414-7433 

The  Casey  Family  Partners, 
Spokane,  Wash.,  (509)  473- 

Carol  Seval,  R.N., 
L.M.H.C. 

Christopher  Blod¬ 
gett,  Ph.D. 

Carol  Seval,  R.N., 

L. M.H.C. 

Mary  Ann  Murphy, 

M. S. 

Ruth  Rose-Jacobs, 
Sc.D. 

Christopher  Blod¬ 
gett,  Ph.D. 

4810 


'One  of  the  original  SESS  sites  was  unable  to  continue  with  the  study  but  was  an  important  contributor  to 
the  original  design  and  implementation  of  the  project. 
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Table  A.  1 — continued 


Study  Site 

Principal 

Investigator 

Project  Director 

Local  Researcher 

University  of  Miami,  Miami, 

Connie  E.  Morrow, 

K.  Lori  Hanson, 

Emmalec  S.  Band- 

Fla.,  (305)  243-2030 

Ph.D. 

Ph.D. 

stra,  M.D. 

April  L.  Vogel,  Ph.D. 

University  of  Missouri, 

Carol  J.  Evans, 

Robyn  S.  Boustead, 

Carol  J.  Evans, 

Columbia,  Mo.,  (573)  884- 

Ph.D. 

M.P.A. 

Ph.D. 

2029 

University  of  New  Mexico, 

Andy  Hsi,  M.D., 

Bebeann  Bouchard, 

Richard  Boyle, 

Albuquerque,  N.M.,  (505)  272- 
3469 

M.P.II. 

M.Ed. 

PhD. 

Early  Childhood  Sites 

Asian  American  Recovery 

Davis  Y.  Ja,  Ph.D. 

Anne  Morris,  Ph.D. 

Anne  Morris,  Ph.D. 

Services,  Inc.,  San  Francisco, 

Calif.,  (415)  541-9285,  ext.  227 

Child  Development  Inc.,  Rus- 

JoAnn  Williams, 

Carol  Amundson 

Mark  C.  Edwards, 

sellville,  Ark.,  (501)968-6493 

M.Ed. 

Lee,  M.A.,  L.P.C. 

Ph.D. 

Children’s  National  Medical 

Jill  G.  Joseph,  M.D., 

Amy  Lewin,  Psy.D. 

Michelle  J.  C.  New, 

Center,  Washington,  D.C., 

Ph.D. 

Ph.D. 

(202)  884-3106 

Johns  Hopkins  University,  Bal¬ 
timore,  Md„  (410)  955-3989 

Philip  J.  Leaf,  Ph.D. 

Jocelyn  Turner- 
Musa,  Ph.D. 

Philip  J.  Leaf,  Ph.D. 

Division  of  Child  and  Family 

Christa  R.  Peterson, 

Laurel  Swetnam, 

Margaret  P.  Freese, 

Services,  Las  Vegas,  Nev., 

Ph.D. 

M.A.,  M.S. 

Ph.D.,  M.P.H. 

(702)  486-6147 

The  Tulalip  Tribes  Beda?chelh, 

Linda  L.  Jones,  B.A. 

Linda  L.  Jones,  B.A. 

Claudia  Long,  Ph.D. 

Marysville,  Wash.,  (360)  651- 
3282 

The  Women’s  Treatment  Cen- 

Jewell  Oates,  Ph.D. 

Dianne  Stansberry7, 

Victor  J.  Bernstein, 

ter,  Chicago,  Ill.,  (773)  373- 

B.A.,  C.S.A.D.P. 

Ph.D. 

8670,  ext.  302 
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Nelba  Chavez,  Ph.D.  Ruth  Massinga,  M.S. 

Administrator,  SAMHSA  President  and  CEO,  Casey 

Rockville,  Maryland  Family  Programs 

Seattle,  Washington 

along  with  the  Casey  Board  of  Trustees  and  the  three  SAMHSA  Centers — Center  for 
Substance  Abuse  Prevention,  Center  for  Substance  Abuse  Treatment,  and  Center  for 
Mental  Health  Services — for  their  vision  and  commitment  to  reaching  families  with 
very  young  children  affected  by  environments  of  substance  abuse  and  mental  dis¬ 
orders.  Without  their  innovative  public-private  partnership  and  unprecedented  sup¬ 
port,  this  initiative  would  have  been  impossible. 
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Joe  Autry,  M.D. 

Acting  Administrator 
SAMHSA 

Pat  Salomon,  M.D. 
Michele  Basen,  M.P.A. 
Velva  Springs,  M.S.W. 
Jocelyn  Whitfield,  M.A. 


Jean  McIntosh,  M.S.W. 

Executive  Vice  President 

Casey  Strategic  Planning  and  Development 

Barbara  Kelley  Duncan,  M.Ed. 

Peter  Pecora,  Ph.D. 

Eileen  O’Brien,  Ph.D. 
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MISSION  STATEMENTS  OF  THE  NATIONAL  COLLABORATORS 


SUBSTANCE  ABUSE  AND  MENTAL  HEALTH  SERVICES  ADMINISTRATION 
(SAMHSA) 

SAMHSA’s  mission  within  the  nation’s  health  system  is  to  improve  the  quality  and 
availability  of  prevention,  treatment,  and  rehabilitation  services  to  reduce  illness, 
death,  disability,  and  cost  to  society  resulting  from  substance  abuse  and  mental  ill¬ 
ness. 

SAMHSA’s  mission  is  accomplished  in  partnership  with  all  concerned  with  substance 
abuse  and  mental  illness.  SAMHSA  exercises  leadership  in 

•  eliminating  the  stigma  that  impedes  prevention,  treatment,  and  rehabilitation 
services  for  individuals  with  substance  abuse; 

•  developing,  synthesizing,  and  disseminating  knowledge  and  information  to 
improve  prevention,  treatment,  rehabilitation  services,  and  improving  the  orga¬ 
nization,  financing,  and  delivery  of  these  services; 

•  providing  strategic  funding  to  increase  the  effectiveness  and  availability  of  ser¬ 
vices; 

•  promoting  effective  prevention,  treatment,  and  rehabilitation  policies  and  ser¬ 
vices; 

•  developing  and  promoting  quality  standards  for  service  delivery; 

•  developing  and  promoting  models  and  strategies  for  training  and  education; 

•  developing  and  promoting  useful  and  efficient  data  collection  and  evaluation 
systems;  and 

•  promoting  public  and  private  policies  to  finance  prevention,  treatment,  and 
rehabilitation  services  so  that  they  are  available  and  accessible. 

For  more  information,  visit  SAMHSA’s  Web  site  at  www.SAMHSA.gov. 

CASEY  FAMILY  PROGRAMS 

The  mission  of  Casey  Family  Programs  is  to  support  families,  youth,  and  children  in 
reaching  their  full  potential.  Casey  provides  an  array  of  permanency  planning,  pre- 
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vention,  and  transition  services,  such  as  long-term  family  foster  care,  adoption,  kin¬ 
ship  care,  job  training,  and  scholarships. 

The  program  aims  to  improve  public  and  private  services  for  children,  youth,  and 
families  impacted  by  the  child  welfare  system,  through  advocacy  efforts,  national 
and  local  community  partnerships,  and  by  serving  as  a  center  for  information  and 
learning  about  children  in  need  of  permanent  family  connections. 

Casey  Family  Programs  is  a  Seattle-based  private  operating  foundation,  established 
by  Jim  Casey,  founder  of  United  Parcel  Service  (UPS),  in  1966.  The  program  has  29 
offices  in  14  states  and  Washington,  D.C.  For  more  information,  visit  our  Web  site  at 
www.casey.org. 
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Starting  Early  to  Link  Enhanced  Comprehensive  Treatment  Teams  ]\jew  fy[exico 

Casey  Family  Partners  Washington 

National  Association  for  Families  and  Addiction  Research  and  Education  Tllinmc* 


Child  Development,  Inc. 

Asian  American  Recovery  Services,  Inc. 
Locally  Integrated  Services  in  Head  Start 


Starting  Early  Starting  Smart  Head  Start  Collaboration  Project  Illinois 

Baltimore  BETTER  Family  and  Community  Partnership  Maryland 
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Evaluation,  Management  and  Training,  Inc/'*  California 


*One  of  the  original  SESS  sites  was  unable  to  continue  with  the  study,  but  it  was  an  important  contributor  to  the 
original  design  and  implementation  of  this  project.  Our  thanks  to  Dr.  Linda  Randolph  and  Dr.  Ira  Chasnoff. 
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Please  feel  free  to  be  a  “copy  cat”  by  making  all  the  copies  you  want  of  the  entire 
document;  or  if  sections  are  copied,  please  provide  the  full  citation  to  the  report. 


For  further  information 
on  alcohol,  tobacco, 
and  illicit  drugs, 
call  1-800-729-6686, 
301-468-2600,  or  TDD 
1-800-487-4889. 
Or  visit  the  World 
Wide  Web  at 
www.health.org. 
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